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Abstract: Within the framework of the Testing Algorithmic and Application Skills project we 
tested first year students of Informatics at the beginning of their tertiary education. We were 
focusing on the students’ level of understanding in different programming environments. In the 
present paper we provide the results from the University of Debrecen, the Universitatea Babes- 
Bolyai, and the Sapientia Hungarian University of Transylvania in two tasks, which are based on 
the same algorithm but presented in different forms. We have found that students in Hungary tend 
to focus on the coding details while with the students in Romania there are only minor differences 
in the results of the two forms. This difference can be explained by the different acceptance 
methods of the universities in the two countries and/or by the methods applied in primary and 
secondary education to solve computer related problems. We can conclude that the approaches 
adopted by students of Informatics in Hungary may be an explanation for the high attrition rates in 
tertiary Computer Science education. 
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1. Introduction 

The first testing period of the Testing Algorithmic and Application Skills (TAaAS) project - the 
academic years 2011/2012-2013/2014 - revealed that first year students of Informatics in Hungary 
starting their tertiary education “...can only think in algorithms in traditional programming 
environments. They do not regard the recent developments in IT as being driven by algorithms.” [1], 
We have also concluded that “.. .methods must be developed and introduced in primary and secondary 
education, which focus on the development of the students’ algorithmic skills in different 
environments and in different computer related activities; the development of students’ algorithmic 
skills should be independent of computer environments and computer related activities.” [1]—[12]. The 
analyses of the students’ results have also revealed that their level of algorithmic skills hardly exceeds 
that of 5th-8th grade students ([1], [13], Figure 1). 

In the subsequent phase of the TAaAS project we invited universities outside Hungary to participate in 
the testing of first year students of Informatics at the beginning of their tertiary education. In the 
2014/2015 academic year six further countries joined the project: Romania, the Czech Republic, 
Slovakia, Serbia, Poland, and France. In the present paper we provide the results of the Faculty of 
Informatics, University of Debrecen, Hungary and the two universities from Romania - Universitatea 
Babes-Bolyai and Sapientia Hungarian University of Transylvania. The specialty of these universities 
is that they accept Hungarian minorities. The focus of this testing period is on how students interpret 
counting problems presented in different forms. 

2. Testing process 

The results of two tasks of the 2014/2015 test are compared in the present paper. The algorithm of the 
two tasks is the same; however, their implementations are different. Both of the tasks are simple 
counting algorithms. The input array and the variable which contains the result are provided in both 
cases. The difference between the two tasks was the presentation of the problem. In one case a pseudo 
code was provided (Figure 1) [11], [12], while in the other a flowchart with smiles functioned as the 
items of the input vector (Figure 2). 
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The pseudo code is borrowed from the first round of the 2008/2009 Hungarian Programming Contest 
for 5th-8th-grade students [13]. The flowchart originated in the TAaAS project. With the smiles and 
the flowchart, our aim was to present the algorithm in a form which is closely connected to the 
graphical programming enironments which are preferred in primary and secondary education. 

What does the following program do? What is stored in variable DB? 



N=50, measured values: 500. 500. 500, 500. 500, 600, 600, 650, 700. 750, 820. 880. 930, 1010, 1050, 980. 930, 
830. 780, 720. 720. 710, 700. 750, 770. 790, 820, 880. 880, 820. 760, 740. 600. 500, 560. 670, 780, 820, 920, 
880. 860, 820. 770. 770, 760. 750, 740. 740, 730, 720. 

DB:=0 

Loop from i= 1 to N 
If X(i)>800 then DB:=DB+1 
End loop 


Figure 1. The pseudo code of a counting algorithm 



Figure 2. The flowchart of a counting algorithm 


3. The sample 

The three universities offer similar courses in Informatics. However, there are differences which we 
have to take into consideration. First of all, the acceptance methods are different in the two countries. 
In Hungary, students are accepted to the university based on their results in the high school graduation 
process. The calculating method is detailed in [14], Here, we must note that entering higher education 
courses in Informatics does not require any knowledge or any exam qualification in Informatics. Even, 
if students take a graduation exam in Informatics, they do not have to complete programming tasks. At 
intermediate level there is no such task, while at advanced level it can be ignored. On the other hand, 
in Romania there is an entrance exam in Informatics or Mathematics at several universities, which 
focuses on the students’ knowledge in problem solving. The Universitatea Babes-Bolyai accepts 
students based on their written entrance exam, either in Informatics or Mathematics, or their results in 
UBB Math-Informatics competitions [15], while the acceptance method at the Sapientia Hungarian 
University of Transylvania is similar to the Hungarian system, based on the results of the high school 
graduation exam [16]. We must also note here that the most popular university in Romania offering 
courses in Hungarian is the Universitatea Babes-Bolyai, while courses in Romanian are available at 
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several other institutes around the country [17], [18]. Consequently, majority students have more 
options in selecting a tertiary institute than minority students [18]. 

The other obvious difference between the institutes in Hungary and Romania is the language. In the 
tested universities in Romania, Hungarian is the minority language; however, they offer courses in 
Romanian, which is the majority language of the country, and also in English and German. We must 
note here that the University of Debrecen also offers courses in Informatics in English. However, the 
foreign language courses are attended by students who arrive mostly from underdeveloped countries in 
Africa and Asia, whose backgrounds are not known; for these reasons they are not included in this 
phase of the test. 

The test was completed by all the different courses and sites at the universities. At the University of 
Debrecen four courses were tested: Software Engineering (UD_SOE), System Engineering 
(UD_SYE), Business Informatics (UD_BI), and Library and Information Sciences (UD_LIS). At the 
Universitatea Babes-Bolyai students were distinguished by the language in which they study. Based on 
the languages, four groups of the students are identified: Hungarian (UBB_HU), Romanian 
(UBB_RO), English (UBB_EN), and German (UBB_GE). At the Sapientia Hungarian University of 
Transylvania (Sapientia University, SU), the sites of the campuses was different. Here we were able to 
distinguish two groups: one in Targu Mures (SU_TM) and one in Miercurea Ciuc (SU_MC). Table 1 
presents the number of students participating in the test and their results in the decoding of the pseudo 
code (PSC) and the flowchart (FCH) tasks. 


Table 1. The number of students involved in the present phase of the TAaAS project and their average results 
based on the SOLO categories of understanding [ 19]—[21] 



Sample 

Pseudo code(PSC) 

Flowchart (FCH) 


University of Debrecen (UD) 

UD_SOE 

117 

3.62 

2.34 

UD_SYE 

89 

2.65 

1.93 

UD_BI 

86 

1.66 

1.30 

UD_LIS 

9 

1.00 

1.11 


Universitatea Babes-Bolyai (UBB) 

UBB_HU 

105 

4.68 

4.17 

UBB_RO 

295 

2.36 

1.72 

UBB_EN 

166 

2.39 

1.71 

UBB_GE 

29 

1.00 

1.00 


Sapientia Hungarian University of Transylvania (SU) 

SU_TM 

136 

2.58 

2.18 

SU_MC 

45 

1.26 

1.15 
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Figure 3. The frequency of the results of the students in the two counting tasks where the evaluation is based on 

the SOLO categories of understanding [19]—[21] 



task O PSC -! FCH 


Figure 4. The average results of the students in the two counting tasks 


4. The evaluation 

To evaluate the students’ answers we used the SOLO categories of understanding [ 19]—[21]. Matching 
these categories to our tasks we were able to distinguish five categories. 

- Ignored (1), those who did not work on the task. 
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- Prestructural (2), mainly those students who focused on the details of the language and loop, 
instead of the problem. 

Unistructural (3), those who considered only one aspect of the problem. 

- Multistructural (4), those who understood more than one aspect but were not able to form a 
complete answer. 

- Relational (5), those students who gave a correct answer. 

Due to the circumstances of the test, the extended abstract SOLO category is left out, since it has no 
relevance [1]. 

Based on the SOLO results we were able to calculate the averages of the students’ results in the 
different groups (Table 1, Figure 3-4 and Table 7, Figure 5-6). In the preliminary analysis the 
averages and the confidence intervals revealed that in the UD_BI, UD_LIS, UBB_GE, and SU_MC 
courses most students were not able to complete either of the tasks. Consequently, we were not able to 
compare their results in the two tasks, and they are left out of the detailed analyses. From this point on 
the following five groups were analyzed: UD_SOE, UD_SYE, UBB_HU, UBB_RO, and SU_TM. 

5. Hypotheses 

The previous analyses of the results of the TAaAS project have already suggested that students of 
Informatics in Hungary start their tertiary education with a lower level of computational thinking and 
algorithmic skills than it is expected by university standards. Based on these results we formed our 
hypotheses, expecting that the international testing would reveal some reasons for this discrepancy. 

HI: There is a connection between the two tasks. We have the methods and the tools to measure the 
results of the algorithm presented in the two different forms. 

H2: There is a significant difference between the results. The result of the FCH task is better than the 
PSC. The FCH form of the algorithm fulfils the spatial-visual-stimuli-preference of students in the 
digital age. 

H3: There is a significant difference between the groups in both tasks. 

6. Results 

6.1. All the participating students 

The results of the different institutes in the two tasks were compared with the Kruskal-Wallis test. 
First of all, it was found that there is a significant difference between the analyzed groups in both tasks 
(in both Kruskal-Wallis tests the group served as the independent variable and the PSC and the FCH 
as the dependent variable in each test, p<0.001). The order of the institutes, based on the results, is 
similar in the two tasks. In both tasks the best results are from the Hungarian students of the 
Universitatea Babes-Bolyai (UBB_HU), followed by the SOE students of the University of Debrecen 
(UD_SOE), with the Romanian students of the Universitatea Babes-Bolyai (UBB_RO) coming last. In 
the third and fourth position, however, there is a change between the SYE students of the University of 
Debrecen (UD_SYE) and the TM students of the Sapientia University (SU_TM). 

The Kruskal-Wallis post hoc test revealed that the results of the UBB_HU group are different from all 
the other groups. The UD_SOE group is also different from the other groups. However, there were no 
significant differences found between the other three groups. In the FCH task another pattern was 
recognized. The result of the UBB_HU is different from all the other groups. The UBB_RO students, 
who have the lowest results in this task, differ from the second and third best - UD_SOE and 
SU_TM students - but not from the UD_SYE group. In no other cases was any significance difference 
found. 

It is also clear from Table 1 and Figures 3 and 4 that the results of decoding the pseudo code (PSC) are 
higher than the flowchart (FCH) in all the analyzed groups. However, it is remarkable that the 
difference between the two tasks of the UD_SOE students is much higher than that of other groups. A 
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similar tendency can be found in the UD_SYE group, which explains the switch in the third and fourth 
positions in the comparison of the two tasks (Figure 4). 

Following that we compared the five groups, considering the differences in the results of the two tasks. 
In this respect, significant differences were also found between the groups (Kruskal-Wallis test, 
p<0.001, Table 2 and 3). 


Table 2. The results of the multiple comparison test after Kruskal-Wallis in the PSC task 


Pairs of groups 

Observed diff. 

Critical diff. 

Is it significant? 

UD_SOE-UD_SYE 

107.16 

84.63 

TRUE 

UD_SOE-UBB_RO 

132.47 

65.74 

TRUE 

UD_SOE-UBB_HU 

115.37 

80.88 

TRUE 

UD_SOE-SU_TM 

114.72 

75.87 

TRUE 

UD_SYE-UBB_RO 

25.31 

72.76 

FALSE 

UD_SYE-UBB_HU 

222.53 

86.69 

TRUE 

UD_SYE-S U_T M 

7.56 

82.03 

FALSE 

UBB_RO-UBB_HU 

247.85 

68.37 

TRUE 

UBB_RO-SU_TM 

17.75 

62.36 

FALSE 

UBB_HU-SU_TM 

230.10 

78.16 

TRUE 


Table 3. The results of the multiple comparison test after Kruskal-Wallis in the FCH task 


Pairs of groups 

Observed diff. 

Critical diff. 

Is it significant? 

UD_SOE-UD_SYE 

40.12 

84.63 

FALSE 

UD_SOE-UBB_RO 

79.76 

65.74 

TRUE 

UD_SOE-UBB_HU 

210.81 

80.88 

TRUE 

UD_SOE-S U_T M 

16.00 

75.87 

FALSE 

UD_SYE-UBB_RO 

39.64 

72.76 

FALSE 

UD_SYE-UBB_HU 

250.92 

86.69 

TRUE 

UD_SYE-S U_T M 

24.10 

82.03 

FALSE 

UBB_RO-UBB_HU 

290.57 

68.372 

TRUE 

UBB_RO-SU_TM 

63.74 

62.36 

TRUE 

UBB_HU-SU_TM 

226.83 

78.16 

TRUE 


Based on the Kruskal-Wallis tests we can conclude that the results of the UD_SOE students are 
significantly different from all the other groups, except for the SOE_SYE group. 

The distribution of the SOEO results in both tasks suggests a merging of the middle three categories 
(Figure 3). By merging the three middle categories we arrive at a three-phase scale: 

Ignored (I) 

- Partially correct (P) 

- Correct (C) 

With the three-phase scale we can carry out a loglinear analysis of the connections among the results 
of the two tasks. The variables considered are the groups (G) (5), task PSC (3), and task FCH (3). 

Based on the corresponding 5H3H3 contingency table, the following loglinear model fits our data: 
GHPSC GHFCH PSCHFCH, Khi-squared(16)=l 1.59, p=0.7714. 

The variables are connected considering the pairs; however, there is no ternary interaction. The lack of 
ternary interaction means that the connection between tasks PSC and FCH does not differ significantly 
within the groups. We must emphasize here that this result does not contradict the results mentioned 
previously, since there it was not the connection between tasks PSC and FCH which was tested, but 
the differences in the results. 

The connections among the pairs revealed by the loglinear - model are shown by the parameters of the 
model, but the results can clearly be seen in the cross-reference tables (Tables 4-6). 
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There is a strong connection between the two forms of the algorithm, which is independent of the 
groups. Table 4 shows that with an increase in the results of the PSC task an increase can also be 
detected in the results of the FCH task. We can conclude that solving the two tasks requires similar 
skills. 


Table 4. The crossreference table of the three-phased solutions of the two tasks 


^\FCH 

PSC 

I 

P 

C 

Total 

I 

257 

38 

5 

300 

85.7% 

12.7% 

1,7% 

100% 


73 

60 

13 

146 


50% 

41.1% 

8.9% 

100% 


78 

91 

127 

296 

L 

26.4% 

30.7% 

42.9% 

100% 

Total 

408 

189 

145 

742 

55% 

25.5% 

19.5% 

100% 


We can read from Table 4 that there is a high probality that the results in task PSC maximize the 
results in task FCH. Considering the connections between the groups and the results obtained in the 
tasks, we can observe that the result of the UD_SOE group, whose result is the second best in task 
SCP, dropped considerably, so its result equals that of the merged sample. 

Tables 5 and 6 show the connections between the groups and the tasks. In the rows of the table the 
distribution of the results is presented in absolute numbers and in percentages. In the PSC task the 
most frequent category is C in the UD_SOE, UBB_HU groups, while in the FCH task this is only true 
for the UBB_HU group. In all the other groups the most frequent category was I. 


Table 5. The distribution of the results of the PSC task in the five groups. The shaded cells show the highest 
frequency in the groups. 


^\PSC 

G 

I 

P 

C 

Total 

UD SOE 

25 

33 

59 

117 



52.9% 

28.2% 

50.4% 

100% 

UD SYE 

41 

24 

24 

89 


46.1% 

27% 

27% 

100% 

SU TM 

72 

21 

43 

136 

15.4% 


52.9% 

31.6% 

100% 

UBB HU 

4 

11 

90 

105 

3.8% 

10.5% 


85.7% 

100% 

UBB RO 

158 

57 

80 

295 


53.6% 

19.3% 

27.1% 

100% 

Total 

300 

146 

296 

742 

40.4% 

19.7% 

39.9% 

100% 
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Table 6. The distribution of the results of task FCH in the five groups. The shaded cells show the highest 
frequency in the groups. 


'^\FCH 

G 

I 

P 

C 

Total 

UD_SOE 

61 

52.1% 

33 

28.2% 

23 

19.7% 

11 

100% 

UD_SYE 

51 

57.3% 

33 

37.1% 

5 

5.6% 

89 

100% 

SU_TM 

74 

41 

21 

136 

54.4% 

30.1% 

15.4% 

100% 

UBB_HU 

7 

34 

64 

105 

6.7% 

32.4% 

61% 

100% 

UBB_RO 

215 

48 

32 

295 

72.9% 

16.3% 

10.8% 

100% 

Total 

408 

189 

145 

742 

55% 

25.5% 

19.5% 

100% 


6.2. Students who worked with the tasks 

In the following, we present the results of those students who worked with the problems. We must 
note here that we do not have reliable information about the students’ reasons for ignoring the tasks. 
However, informal interviews revealed that there are at least two reasons: they did not know the 
answers, or they did not take the test seriously. 

Table 7. The number of students working with both tasks and their average results based on the SOLO 
categories of understanding 



Sample 

Pseudo code(PSC) 

Flowchart (FCH) 


University of Debrecen (UD) 

UD SOE 

53-45.29% 

4.51 

3.85 

UD SYE 

32 - 35.96% 

4.06 

3.34 


Universitatea Babes-Bolyai (UBB) 

UBB HU 

96-91.43% 

4.84 

4.42 

UBB RO 

61 -20.68% 

4.34 

3.80 


Sapientia Hungarian University of Transylvania (SU) 

SU_TM 

50 - 36.76% 

4.42 

3.90 


Table 7 shows the number of students who worked with both tasks, while Figure 5 illustrates the 
distribution of their SOLO-results, and Figure 6 the average of these results. Ignoring the uncompleted 
tasks, we have a significant difference between the groups in the PSC task (Kruskal-Wallis test, 
p<0.001). However, the comparison of the pairs revealed that this difference is due to the difference 
between the best and the weakest groups, UBB_HU and UD_SYE, respectively. There was no 
significant difference found between the other groups. In task FCH, a significant difference was also 
noted between the groups. In this task, on one hand, similar to task PSC, the best and the weakest 
groups’ results proved to be significantly different. On the other hand, we have found a significant 
difference between the two groups from the UBB, the Hungarian and the Romanian students, 
respectively. 

As was mentioned earlier, these results can be explained by the distribution of the Romanian and the 
Hungarian students in the tertiary institutes in Romania [18]. However, there might be another 
explanation which is strongly supported by the results of research into minorities. This research has 
found that “minority parents ... urge their children to study harder...” “...realizing the importance of 
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higher education studies, they try to do their best in any way they can to support their children in 
becoming successful students.” [22]. 



Figure 5. The frequency of the results of the students who worked with the problems, based on the SOLO 

categories of understanding 



Figure 6. The average results of the students who worked with the problems 

When comparing the differences between the results of the two tasks completed by the groups no 
significant difference was found (Kruskal-Wallis test, p = 0.573). In this sense none of the groups are 
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different from the others. Leaving out those students who did not complete the task reduced the 
differences between the groups. Consequently, the results of the students who did not complete the 
tasks would shed light on the lack of algorithmic skills, problem solving approaches, and attitude. 

7. Revisiting our hypotheses 

In our test, we have presented two tasks based on the same counting algorithm. The only difference 
between the tasks was their appearance. The pseudo code tends to follow the form of coding typical of 
high level programming languages, so there was a choice between the smiles with the flowchart to 
reveal whether the students prefer the spatial-visual stimuli to the code based form [23]. Considering 
our hypotheses we have come to the following conclusions. 

Hypothesis HI was formulated considering the similarities of the two tasks. In accordance with our 
hypotheses, there is a connection between the two tasks, so when an increase is detected in the result 
of the PSC task a similar tendency is found in the FCH task. It was also found that there is a 
significant difference between the two tasks. However, the results are better in the pseudo code than in 
the flowchart task, in all the analyzed groups. Based on this result we must reject the second half of 
our H2 hypothesis. Students of Informatics prefer the form which is closer in appearance to the coding 
used in high level programming languages to the visualized input. When comparing the groups we 
found significant differences between them. 

However, in the two tasks different connections and differences were recognizable. It is clear that the 
Hungarian students of UBB are significantly better in both tasks than the other groups. H3 hypothesys 
is found true. The SOE students are significantly different from all the other groups in the pseudo code 
task, but not in the flowchart task. In this task they are different from the best and the lowest results. A 
similar tendency is detected in the other group of SYE students in Hungary. These results suggest that 
students in Hungary are more code-dependent than students in Romania. They tend to focus more on 
the coding details than on the algorithm. 

Apropos of this circumstance, one can surmise that the better results of the Hungarian students are 
related to the fact that they are an ethnic minority in Romania, and so performing well in a Hungarian- 
language programme of the most prestigious Romanian state university may also mean protecting and 
asserting the national identity of that minority. Their learning habits and knowledge-management can 
perhaps relate to strategies of identity management that Erzsebet Dani discusses in her book on the 
identity dramas and intercultural strategies of Hungarians in Romania [17]. 

8. Conclusion 

In the Testing Algorithmic and Application Skills (TAaAS) project we tested the level of algorithmic 
skills of first year students of Informatics. In the present paper we have provided the results of five 
groups of students from three universities: University of Debrecen, Universitatea Babes-Bolyai and 
Sapientia Hungarian University of Transylvania. 

The tasks of the test were to measure how the students recognize an algorithm if it is presented in two 
different forms. To decide on the level of understanding and students’ algorithmic skills the SOLO 
categories were applied. Both tasks were a counting algorithm, once given in the form of a pseudo 
code with numbers, and in the other case in a flowchart with smiles of different sizes. This latter form 
was selected to prove the theory that students in the digital age prefer spatial-visual stimuli to 
character-based sources. 

The best results were achieved by the Hungarian students of the Universitatea Babes-Bolyai, followed 
by the SOE group of the University of Debrecen, while the worst were the Romanian group of the 
Universitatea Babes-Bolyai. It is remarkable that in all the groups the results were better in the pseudo 
code task than in the flowchart. Significant differences can be detected in the size of the differences. 
This difference caused the change in the order of the results between the SYE students of the 
University of Debrecen and the students of the Sapientia University. The greatest difference between 
the two tasks was found in the SOE students of the University of Debrecen. 
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Based on the results of our analyses we have come to the conclusion that the students of the 
universities in Romania were able to recognize the algorithm in the two different forms better than the 
students in Hungary. The students’ results in Romania did not change significantly when they were 
faced with a non-character based presentation of an algorithm. Using the SOLO categories and the 
comparison of the results of the two tasks we have found the level of understanding of the students in 
Hungary is lower than that of the students in Romania. 

This can be explained by the different acceptance methods of the students and/or the approaches the 
students take to studying programming and problem solving in primary and secondary education. In 
general, we can conclude that students in Romania start their tertiary education in Informatics with a 
higher level of understanding than students in Hungary. 
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